{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "# Structured data classification from scratch\n",
    "\n",
    "**Author:** [fchollet](https://twitter.com/fchollet)<br>\n",
    "**Date created:** 2020/06/09<br>\n",
    "**Last modified:** 2020/06/09<br>\n",
    "**Description:** Binary classification of structured data including numerical and categorical features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "This example demonstrates how to do structured data classification, starting from a raw\n",
    "CSV file. Our data includes both numerical and categorical features. We will use Keras\n",
    "preprocessing layers to normalize the numerical features and vectorize the categorical\n",
    "ones.\n",
    "\n",
    "Note that this example should be run with TensorFlow 2.3 or higher, or `tf-nightly`.\n",
    "\n",
    "### The dataset\n",
    "\n",
    "[Our dataset](https://archive.ics.uci.edu/ml/datasets/heart+Disease) is provided by the\n",
    "Cleveland Clinic Foundation for Heart Disease.\n",
    "It's a CSV file with 303 rows. Each row contains information about a patient (a\n",
    "**sample**), and each column describes an attribute of the patient (a **feature**). We\n",
    "use the features to predict whether a patient has a heart disease (**binary\n",
    "classification**).\n",
    "\n",
    "Here's the description of each feature:\n",
    "\n",
    "Column| Description| Feature Type\n",
    "------------|--------------------|----------------------\n",
    "Age | Age in years | Numerical\n",
    "Sex | (1 = male; 0 = female) | Categorical\n",
    "CP | Chest pain type (0, 1, 2, 3, 4) | Categorical\n",
    "Trestbpd | Resting blood pressure (in mm Hg on admission) | Numerical\n",
    "Chol | Serum cholesterol in mg/dl | Numerical\n",
    "FBS | fasting blood sugar in 120 mg/dl (1 = true; 0 = false) | Categorical\n",
    "RestECG | Resting electrocardiogram results (0, 1, 2) | Categorical\n",
    "Thalach | Maximum heart rate achieved | Numerical\n",
    "Exang | Exercise induced angina (1 = yes; 0 = no) | Categorical\n",
    "Oldpeak | ST depression induced by exercise relative to rest | Numerical\n",
    "Slope | Slope of the peak exercise ST segment | Numerical\n",
    "CA | Number of major vessels (0-3) colored by fluoroscopy | Both numerical & categorical\n",
    "Thal | 3 = normal; 6 = fixed defect; 7 = reversible defect | Categorical\n",
    "Target | Diagnosis of heart disease (1 = true; 0 = false) | Target\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Preparing the data\n",
    "\n",
    "Let's download the data and load it into a Pandas dataframe:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "file_url = \"https://storage.googleapis.com/applied-dl/heart.csv\"\n",
    "dataframe = pd.read_csv(file_url)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "The dataset includes 303 samples with 14 columns per sample (13 features, plus the target\n",
    "label):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "dataframe.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Here's a preview of a few samples:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "dataframe.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "The last column, \"target\", indicates whether the patient has a heart disease (1) or not\n",
    "(0).\n",
    "\n",
    "Let's split the data into a training and validation set:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "val_dataframe = dataframe.sample(frac=0.2, random_state=1337)\n",
    "train_dataframe = dataframe.drop(val_dataframe.index)\n",
    "\n",
    "print(\n",
    "    \"Using %d samples for training and %d for validation\"\n",
    "    % (len(train_dataframe), len(val_dataframe))\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Let's generate `tf.data.Dataset` objects for each dataframe:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "\n",
    "def dataframe_to_dataset(dataframe):\n",
    "    dataframe = dataframe.copy()\n",
    "    labels = dataframe.pop(\"target\")\n",
    "    ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))\n",
    "    ds = ds.shuffle(buffer_size=len(dataframe))\n",
    "    return ds\n",
    "\n",
    "\n",
    "train_ds = dataframe_to_dataset(train_dataframe)\n",
    "val_ds = dataframe_to_dataset(val_dataframe)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Each `Dataset` yields a tuple `(input, target)` where `input` is a dictionary of features\n",
    "and `target` is the value `0` or `1`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "for x, y in train_ds.take(1):\n",
    "    print(\"Input:\", x)\n",
    "    print(\"Target:\", y)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Let's batch the datasets:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "train_ds = train_ds.batch(32)\n",
    "val_ds = val_ds.batch(32)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Feature preprocessing with Keras layers\n",
    "\n",
    "\n",
    "The following features are categorical features encoded as integers:\n",
    "\n",
    "- `sex`\n",
    "- `cp`\n",
    "- `fbs`\n",
    "- `restecg`\n",
    "- `exang`\n",
    "- `ca`\n",
    "\n",
    "We will encode these features using **one-hot encoding** using the `CategoryEncoding()`\n",
    "layer.\n",
    "\n",
    "We also have a categorical feature encoded as a string: `thal`. We will first create an\n",
    "index of all possible features using the `StringLookup()` layer, then we will one-hot\n",
    "encode the output indices using a `CategoryEncoding()` layer.\n",
    "\n",
    "Finally, the following feature are continuous numerical features:\n",
    "\n",
    "- `age`\n",
    "- `trestbps`\n",
    "- `chol`\n",
    "- `thalach`\n",
    "- `oldpeak`\n",
    "- `slope`\n",
    "\n",
    "For each of these features, we will use a `Normalization()` layer to make sure the mean\n",
    "of each feature is 0 and its standard deviation is 1.\n",
    "\n",
    "Below, we define 3 utility functions to do the operations:\n",
    "\n",
    "- `encode_numerical_feature` to apply featurewise normalization to numerical features.\n",
    "- `encode_string_categorical_feature` to first turn string inputs into integer indices,\n",
    "then one-hot encode these integer indices.\n",
    "- `encode_integer_categorical_feature` to one-hot encode integer categorical features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers.experimental.preprocessing import Normalization\n",
    "from tensorflow.keras.layers.experimental.preprocessing import CategoryEncoding\n",
    "from tensorflow.keras.layers.experimental.preprocessing import StringLookup\n",
    "\n",
    "\n",
    "def encode_numerical_feature(feature, name, dataset):\n",
    "    # Create a Normalization layer for our feature\n",
    "    normalizer = Normalization()\n",
    "\n",
    "    # Prepare a Dataset that only yields our feature\n",
    "    feature_ds = dataset.map(lambda x, y: x[name])\n",
    "    feature_ds = feature_ds.map(lambda x: tf.expand_dims(x, -1))\n",
    "\n",
    "    # Learn the statistics of the data\n",
    "    normalizer.adapt(feature_ds)\n",
    "\n",
    "    # Normalize the input feature\n",
    "    encoded_feature = normalizer(feature)\n",
    "    return encoded_feature\n",
    "\n",
    "\n",
    "def encode_string_categorical_feature(feature, name, dataset):\n",
    "    # Create a StringLookup layer which will turn strings into integer indices\n",
    "    index = StringLookup()\n",
    "\n",
    "    # Prepare a Dataset that only yields our feature\n",
    "    feature_ds = dataset.map(lambda x, y: x[name])\n",
    "    feature_ds = feature_ds.map(lambda x: tf.expand_dims(x, -1))\n",
    "\n",
    "    # Learn the set of possible string values and assign them a fixed integer index\n",
    "    index.adapt(feature_ds)\n",
    "\n",
    "    # Turn the string input into integer indices\n",
    "    encoded_feature = index(feature)\n",
    "\n",
    "    # Create a CategoryEncoding for our integer indices\n",
    "    encoder = CategoryEncoding(output_mode=\"binary\")\n",
    "\n",
    "    # Prepare a dataset of indices\n",
    "    feature_ds = feature_ds.map(index)\n",
    "\n",
    "    # Learn the space of possible indices\n",
    "    encoder.adapt(feature_ds)\n",
    "\n",
    "    # Apply one-hot encoding to our indices\n",
    "    encoded_feature = encoder(encoded_feature)\n",
    "    return encoded_feature\n",
    "\n",
    "\n",
    "def encode_integer_categorical_feature(feature, name, dataset):\n",
    "    # Create a CategoryEncoding for our integer indices\n",
    "    encoder = CategoryEncoding(output_mode=\"binary\")\n",
    "\n",
    "    # Prepare a Dataset that only yields our feature\n",
    "    feature_ds = dataset.map(lambda x, y: x[name])\n",
    "    feature_ds = feature_ds.map(lambda x: tf.expand_dims(x, -1))\n",
    "\n",
    "    # Learn the space of possible indices\n",
    "    encoder.adapt(feature_ds)\n",
    "\n",
    "    # Apply one-hot encoding to our indices\n",
    "    encoded_feature = encoder(feature)\n",
    "    return encoded_feature\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Build a model\n",
    "\n",
    "With this done, we can create our end-to-end model:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "# Categorical features encoded as integers\n",
    "sex = keras.Input(shape=(1,), name=\"sex\", dtype=\"int64\")\n",
    "cp = keras.Input(shape=(1,), name=\"cp\", dtype=\"int64\")\n",
    "fbs = keras.Input(shape=(1,), name=\"fbs\", dtype=\"int64\")\n",
    "restecg = keras.Input(shape=(1,), name=\"restecg\", dtype=\"int64\")\n",
    "exang = keras.Input(shape=(1,), name=\"exang\", dtype=\"int64\")\n",
    "ca = keras.Input(shape=(1,), name=\"ca\", dtype=\"int64\")\n",
    "\n",
    "# Categorical feature encoded as string\n",
    "thal = keras.Input(shape=(1,), name=\"thal\", dtype=\"string\")\n",
    "\n",
    "# Numerical features\n",
    "age = keras.Input(shape=(1,), name=\"age\")\n",
    "trestbps = keras.Input(shape=(1,), name=\"trestbps\")\n",
    "chol = keras.Input(shape=(1,), name=\"chol\")\n",
    "thalach = keras.Input(shape=(1,), name=\"thalach\")\n",
    "oldpeak = keras.Input(shape=(1,), name=\"oldpeak\")\n",
    "slope = keras.Input(shape=(1,), name=\"slope\")\n",
    "\n",
    "all_inputs = [\n",
    "    sex,\n",
    "    cp,\n",
    "    fbs,\n",
    "    restecg,\n",
    "    exang,\n",
    "    ca,\n",
    "    thal,\n",
    "    age,\n",
    "    trestbps,\n",
    "    chol,\n",
    "    thalach,\n",
    "    oldpeak,\n",
    "    slope,\n",
    "]\n",
    "\n",
    "# Integer categorical features\n",
    "sex_encoded = encode_integer_categorical_feature(sex, \"sex\", train_ds)\n",
    "cp_encoded = encode_integer_categorical_feature(cp, \"cp\", train_ds)\n",
    "fbs_encoded = encode_integer_categorical_feature(fbs, \"fbs\", train_ds)\n",
    "restecg_encoded = encode_integer_categorical_feature(restecg, \"restecg\", train_ds)\n",
    "exang_encoded = encode_integer_categorical_feature(exang, \"exang\", train_ds)\n",
    "ca_encoded = encode_integer_categorical_feature(ca, \"ca\", train_ds)\n",
    "\n",
    "# String categorical features\n",
    "thal_encoded = encode_string_categorical_feature(thal, \"thal\", train_ds)\n",
    "\n",
    "# Numerical features\n",
    "age_encoded = encode_numerical_feature(age, \"age\", train_ds)\n",
    "trestbps_encoded = encode_numerical_feature(trestbps, \"trestbps\", train_ds)\n",
    "chol_encoded = encode_numerical_feature(chol, \"chol\", train_ds)\n",
    "thalach_encoded = encode_numerical_feature(thalach, \"thalach\", train_ds)\n",
    "oldpeak_encoded = encode_numerical_feature(oldpeak, \"oldpeak\", train_ds)\n",
    "slope_encoded = encode_numerical_feature(slope, \"slope\", train_ds)\n",
    "\n",
    "all_features = layers.concatenate(\n",
    "    [\n",
    "        sex_encoded,\n",
    "        cp_encoded,\n",
    "        fbs_encoded,\n",
    "        restecg_encoded,\n",
    "        exang_encoded,\n",
    "        slope_encoded,\n",
    "        ca_encoded,\n",
    "        thal_encoded,\n",
    "        age_encoded,\n",
    "        trestbps_encoded,\n",
    "        chol_encoded,\n",
    "        thalach_encoded,\n",
    "        oldpeak_encoded,\n",
    "    ]\n",
    ")\n",
    "x = layers.Dense(32, activation=\"relu\")(all_features)\n",
    "x = layers.Dropout(0.5)(x)\n",
    "output = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "model = keras.Model(all_inputs, output)\n",
    "model.compile(\"adam\", \"binary_crossentropy\", metrics=[\"accuracy\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "Let's visualize our connectivity graph:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "# `rankdir='LR'` is to make the graph horizontal.\n",
    "keras.utils.plot_model(model, show_shapes=True, rankdir=\"LR\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Train the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "model.fit(train_ds, epochs=50, validation_data=val_ds)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "We quickly get to 80% validation accuracy.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## Inference on new data\n",
    "\n",
    "To get a prediction for a new sample, you can simply call `model.predict()`. There are\n",
    "just two things you need to do:\n",
    "\n",
    "1. wrap scalars into a list so as to have a batch dimension (models only process batches\n",
    "of data, not single samples)\n",
    "2. Call `convert_to_tensor` on each feature\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "sample = {\n",
    "    \"age\": 60,\n",
    "    \"sex\": 1,\n",
    "    \"cp\": 1,\n",
    "    \"trestbps\": 145,\n",
    "    \"chol\": 233,\n",
    "    \"fbs\": 1,\n",
    "    \"restecg\": 2,\n",
    "    \"thalach\": 150,\n",
    "    \"exang\": 0,\n",
    "    \"oldpeak\": 2.3,\n",
    "    \"slope\": 3,\n",
    "    \"ca\": 0,\n",
    "    \"thal\": \"fixed\",\n",
    "}\n",
    "\n",
    "input_dict = {name: tf.convert_to_tensor([value]) for name, value in sample.items()}\n",
    "model.predict(input_dict)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "This particular patient had a 31% probability of having a heart disease, as evaluated by\n",
    "our model.\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "structured_data_classification_from_scratch",
   "private_outputs": false,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}